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Novel signatures of the Higgs sector from S3 flavor symmetry 
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In an earlier work we analyzed the CP-even scalar sector of an S3 flavor model, where we identified 
some novel decay signatures of an exotic scalar. In this work we extend our analysis by including 
the complete set of scalars/pseudoscalars, revisiting the potential minimization conditions in a more 
general setup, setting the spectrum in conformity with the current LHC limits on the scalar mass, 
and identifying yet another spectacularly novel decay channel which might be revealed from an 
intense study of rare top decays at the LHC into modes containing multileptons of different flavors. 
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Discrete flavor symmetries constitute an effective way of explaining the masses and mixing of quarks and leptons [Ij . 
These symmetries may be broken at a high scale by vacuum expectation values of scalars, called flavons or familons 
(see e.g., Ref. 0). Interesting experimental signatures such as nonstandard decays involving scalars and gauge bosons 
and/or large flavor changing neutral currents often arise in such scenarios. The flavor group S3 specifically was 
i - 1 introduced early in Ref. 3J and has since been used in many different flavor scenarios [4] . Our analysis is based on the 
'■^ flavor symmetry realization introduced in Ref. [5] to describe charged lepton and neutrino masses and mixing. The 
1" 1 group structure of S3 favors a maximal atmospheric mixing angle which still makes it a good fit even after the recent 
1 measurements of a nonzero #13 [5]. The group S3 has three irreducible representations: 1, 1', and 2. The invariants 
1 can be constructed using the multiplication rules 2 £g) 2 = 1 © 1' © 2 and 1' <8> 1' = 1- We follow the particle 
assignments [5] as we have in Ref. [7] : 



(L^, L T ) e 2 , L e ,e c ,fi c el, r c £l' ; 



> (Q 2 ,Q 3 )e2, Q 1: u c ,c c ,d c , s c e 1, b c ,t c ei', (1) 

(0!,0 2 )G2, 03 el. 

The fields Cq/2/3 and L e / Il / T refer to the quark and lepton SU(2) doublets of the three generations. This assignment 
was motivated in Ref. [5] in order to have a reasonably successful reproduction of quark and lepton masses and 
^sO mixing. An intuitive appreciation of large mixing in the lepton sector vis-d-vis small mixing for quarks will be 
clear when we discuss the Yukawa Lagrangian in Sec. Ill In this paper we concentrate on the scalar SU(2) doublets 



0{i,2.3} 1 all °f which take part in electroweak symmetry breaking. The general structure of the model allows for 
tree-level flavor changing neutral currents due to the absence of natural flavor conservation [8 , although those are 
sufficiently suppressed by the Yukawa couplings in this model, and the effects remain small even for scalar masses 
of the electroweak scale [3 |H| • However, for models in which the Yukawa couplings are not restricted by any flavor 
symmetry, scalar masses are pushed to the TeV scale [10] . In our previous analysis [7] we found that the additional 
CP-even scalars have noteworthy properties. In this work we observe analogous properties to hold for pseudoscalars 
as well. These properties dictate the main collider signatures: 

1. Two of the three scalars hb, c have standard model (SM)-like couplings except that they can dominantly decay 
into the third scalar h a , whose couplings are not SM-like. 

2. The scalar (pseudoscalar) h a (x a ) has no (h a /x a )VV-type interactions, where V = W ± ,Z. 

3. h a /Xa has only flavor off-diagonal Yukawa couplings with one fermion of the third generation. 

More specifically, we have extended our previous analysis by including not only the CP-even neutral scalars, but 
all scalar degrees of freedom: three CP-even neutral scalars, two CP-odd neutral scalars and two sets of charged 
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scalars. The present analysis extends on our previous discussion^ in the following crucial points: i) determination 
of the mass spectrum of the neutral scalars/pseudoscalars and the charged scalars following an improved potential 
minimization technique, ii) calculation of their couplings to the gauge bosons and matter fields, and Hi) identification 
of a novel channel of a scalar (pseudoscalar) decay within reach of the LHC. 

II. MASS SPECTRUM OF THE SCALARS/PSEUDOSCALARS 

The general S3 invariant scalar potential used is given by Refs. [7J QT] 

V = m 2 (4^ + 4<fe) + mt4h + y + 4^) 2 + y (4^1 ~ 4fa) 2 + ^%4i<t>i + y (<^a) 2 

+ x 5 (4 3 h)(4i^i + 4 2 <h) + ^4 3 (<Pi4i + h4 2 )4>3+ ^4<l>i4<h + >*4(<l>i4fa + fc4<h)+i i -c- . (2) 



Once the scalars receive vacuum expectation values, the replacement </>j —> (hf , Vi + hi + for i = 1 . . . 3 is 

performed, with the assignments v\ = v 2 = v and V3, which allow for maximal atmospheric mixing. To generate 
the correct W ± and Z masses, 2v 2 + u| = u| M has to hold, where v$m = 246 GeV. After diagonalizing the mass 
matrices the masses of the physical scalars/pseudoscalars are obtained. These are denoted by h a ,b,c, Xa,b and h^ b . 
The remaining degrees of freedom are neutral (G°) and charged (G ± ) Goldstone bosons which are eaten up by Z and 
W ± respectively. 

Note that v\ — v 2 is an extremal point if the following conditions are imposed [7] : 

-m 2 = (2Ai + X 3 )v 2 + (A 5 + A 6 + \ 7 )v 2 3 + 3A 8 w 3 , -m\ = A 4 v 2 + 2(A 5 + A 6 + A 7 )v 2 + 2A 8 w 3 /«3 ■ (3) 

To make sure that the extremal point is actually a minimum of the potential, the determinant of the Hessian has 
to be positive. This statement is equivalent to imposing the condition of positive squared masses of the particles. 
To keep the potential globally bounded from below the conventional approach is to arrange all the coefficients of the 
highest-power terms in the potential to be positive definite. This was followed in Ref. [7J where only the CP-even 
degrees of freedom were considered. However, this strategy eliminates the allowed possibility of a large part of valid 
parameter space where the potential is bounded from below although some coefficients still stay negative. 

Our present analysis is now more complete in the sense that we deal with the complete spectrum including all 
neutral and charged degrees of freedom following the potential minimization. Moreover, some parts of the allowed 
parameter space that were hitherto cut off by the traditional method are now resurrected by our new approach. As a 
first step, to have an analytical feel we identify some simple-looking relations of the coefficients by inspection that allow 
the potential to stay positive and also provide the physical scalar masses. To do this the scalar potential in Eq. ^ 
is factorized into a simplified polynomial in 4>i,4>2 and ^3, treating them naively as real quantities for calculational 
ease. There remain three distinct types of terms of order four: cj)f, 4>14 an d 4 ) 1 ( t ) i ( t ) k, where k = 1 . . .3. Out of 
the nine terms, only six have independent coefficients, called c/x...6} : 

ci4 + ci4 + c 2 03 + c 3 44 + Ci44 + c i44 + c^lfofo + c^i^ifo + c^i^l . (4) 

The coefficients c/x...e} can be expressed in terms of the potential parameters A/i,..gi.: 

ci = Ai/2 + A 2 /2, c 2 = A4/2, c 3 = \ 1 -\ 2 + A 3 , c 4 = A s + A 6 , c 5 = 2A 8 , c 6 = 2A 7 . (5) 

By inspection, we found the following conditions on the coefficients 6 j from the analytic expressions: 

Ci > 0, c 2 > 0, 2c3 > —c%, 2c 3 > —c 2 , 2c4 > — ci, 2c 4 > — c 2 , 

— l/2ci < c 5 < ci, — l/2ci < c 6 < ci, -l/2c 2 < c 5 < c 2 , -l/2c 2 < c 6 < c 2 . 



(6) 



These conditions ensure an acceptable mass spectrum for the neutral scalars/pseudoscalars and charged scalars and 
keep the potential globally stable. However, this method renders a large part of the parameter space still inacces- 
sible; moreover, the masses obtained by employing Eq. ([6| are generally quite light, none exceeding 300 GeV when 

|A{1... 8 }| < 7T. 

To obtain a more complete picture we have transformed Eq. Q into spherical coordinates (p,9,<f>). The potential 
then splits into a radial and an angular part. The question of global stability is then reduced to keeping the over-all 
sign of the angular part of the potential positive definite: 

sin 4 0{(2ci - c 3 ) cos(40) + 6c x + c 3 } + 8c 2 cos 4 6 + sin 2 (20) (2c 4 sin 2 <f> + c 6 sin(20)) 

+ 8c 4 cos 2 4> sin 2 9 cos 2 9 + 4c 5 sin(2</>) sin 3 9 cos 9 (sin <f> + cos <j>) > (7) 
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m ha [GeV] m hb [GeV] m hb [GeV] 



FIG. 1. Scatter plots of masses of h a , h b , h c and \a, where v-j/v — 0.6. The lines give the current interesting window between 
114 GeV (LEP2) and 130 GeV (LUC) TM [T^ a) Masses of CP-even scalars h a and h b . b) Mass of CP-even scalar h b plotted 
against the mass difference of h c and h b ; the highlighted strip is also disfavored by LHC which rules out a second SM-like Higgs 
within 550 GeV. c) Mass of h b compared to that of the CP-odd scalar \a- 



As this is a transcendental inequality there is no analytically tractable and simple set of conditions that can be 
imposed on the coefficients C{i...6} to solve Eq. (|7j). We therefore decided to check the positivity of this function 
numerically at each point of the parameter space. This allows us to explore the so-far inaccessible territory of the 
stable parameter space that could not be reached by the conditions of Eq. ffify. Consequently, masses well beyond 
300GeV for the scalars/pseudoscalars can be reached even keeping | A{i... 8 }P < n. To sum up, our Eq. ^ is an 
improvement over what we have done in Ref. [7J and subsequently our numerical approach improves the size of the 
accessible parameter space even further. 

Diagonalizing the mass matrix of the pseudoscalars gives the symmetry basis pseudoscalars xi,2,3 in terms of the 
physical basis pseudoscalars Xa,b- 

Xi(2) = (v/v SM ) G° T (l/v^) Xa - v 3 / (V2v SM ) Xb, X3 = (f Asm) G° + V2 {v 3 /v SM ) Xb ■ (8) 

It is interesting to note that the mixing coefficients are very simple and just depend on the ratio v 3 /vsm- This is in 
stark contrast to the mixing of /ii,2,3 and /i a ,b.c given in Ref. [7J, where the coefficients are complicated functions of 
the A{i...s} parameters of Eq. (|2j): 

^1(2) = ^1(2)6 hf, + £/x(2) c h c =F fl/V2J K, h 3 = U 3b h b + U 3c h c , (9) 

where Ui b and Ui C are complicated functions of the A^ gj, v and v 3 . The mixing relations for the charged scalars 
K^ h are obtained by substituting x ~ > h + and G° — > G + in Eq. pi). The masses for the CP-even scalars are [7] 



m 2 ha = 4A 2 v 2 — 2A 3 u 2 — v 3 (2X 7 v 3 + 5\$v) , 
ml b(c) = ^ [4Aiw 2 w 3 + 2X 3 v 2 v 3 + 2A 4 u| - 2X s v 3 + 3X 8 vv 2 T Am 3 ] , ' ! ' ' 



(11) 



where 

Am 3 = 8vv 3 ^2vv 3 ^2(A 5 + A 6 + A 7 ) 2 - A 4 (2Ai + A 3 )^ + 2A 8 v 4 (2Ai + A 3 ) - 3A 4 A 8 u| 

+ 12A 8 v 2 v 2 (A 5 + A 6 + A 7 ) + 12Xlv 3 v 3 ^ + ^2v 2 v 3 (2X 1 + X 3 ) + 2X 4 v 3 - 2X 8 v 3 + 3A 8 ot 2 

We now derive the pseudoscalar squared masses as 

m 2 Q = -9X s vv 3 , m 2 Xb = -v\ M (2A 7 + X 8 v/v 3 ) . (12) 
The corresponding squared masses for the charged scalars are 

m 2 h + = -2X 3 v 2 - vl (A 6 + A 7 ) + 5A 8 w 3 , m 2 + = -Ug M (A 6 + A 7 + X s v/v 3 ) . (13) 
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The allowed ranges for the masses can be found by a random scattering in the parameter space where the couplings 
in the potential are varied within A{i...g} € [— 7r,7r] and the ratio v 3 /v is fixed to 0.6 (this value was chosen in Ref. [7J 
for compatibility with the quark masses). The allowed range for the couplings A.n ..§} nas been increased with respect 
to what was assumed in our previous work [7J to admit a broader mass spectrum. The values are still very much within 
pertubative bounds. This leads to a CP-even mass spectrum similar to Ref. [7J, but with higher allowed ranges. The 
scalar h a can be as massive as roughly 800 GeV or arbitrarily light as it evades the LEP2 bound due to its maximally 
nonstandard couplings. The mass of the SM-like scalar hb is limited within 114-500 GeV, while h c is still heavier. 
Both hb and h c masses should however satisfy the LEP2 lower bound of 114 GeV [See Fig. [Tjfor details]. 

In view of the recent LHC results [12 |T3] that hint towards a SM-like Higgs boson at around 125 GeV with a large 
excluded region above and below, the mass spectrum in this model is compatible with the following scenario: 

1. hb plays the role of the SM-like Higgs boson with a mass of roughly 125 GeV. The Yukawa and gauge couplings 
of hj, and h c are SM-like with numerically negligible flavor off-diagonal couplings [7J . 

2. The scalars/pseudoscalars h a and \a have nonstandard interactions that hide them from standard searches, as 
will be discussed in the following sections. In particular, h a and % a can be very light. 

3. All other scalar/pseudoscalar masses, including the charged scalars, can be above 550 GeV. We however note 
that the existing limits on charged scalar masses are not so stringent and the parameters of our potential can 
be arranged to admit a much smaller mass for them, though this is not the main focus of our present work. 

III. COUPLINGS OF THE SCALARS/PSEUDOSCALARS 
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TABLE I. Three-point vertices involving at least one neutral scalar/pseudoscalar and gauge bosons. A checkmark indicates that 
the vertex exists. 

It is worth noting that among the couplings listed in Table [I] the ones involving h a do not depend on any parameters 
of the scalar potential, while the couplings of hb and h c to the gauge bosons are complicated functions of the scalar 
mixing parameters, which we refer to in our tables by putting checkmark signs without displaying their explicit forms. 
The haXaZ coupling has a simple form h a \ a Z : — ^Gq^, where G = y/ g 2 + g' 2 and is the momentum transfer. As 
stated in Ref. [7], h a stands out because it does not couple to pairs of gauge bosons via the three-point vertex. As 
a result, neither the LEP2 lower limit of 114 GeV nor the electroweak precision test upper limit of around 200 GeV 
applies to it. The same is true for the pseudoscalar Xa- For certain kinematic regions, the coupling h a x a Z is important 
for collider searches as we shall see later. Table [n] contains the other gauge-scalar-scalar and the triple-scalar vertices. 
Note that h a couples only off-diagonally to the other scalars/pseudoscalars. The h a h^hb couplings depend only on 
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TABLE II. Other three-point vertices. A checkmark indicates that the vertex exists. 

Va/v, while the other triple scalar couplings are complicated functions of the scalar mixing parameters. 

For illustration, we first write the Yukawa Lagrangian of the CP-even scalars in the basis {hi, h 2 , h 3 } with the 
couplings fi for leptons and gi for quarks as in Ref. [7J: 



£y u k = f±ee c h 3 + f 5 e^, c h 3 + /i^ c (/i/i 2 + Th x ) + f 2 T c (-^h 2 + rh x ) + g%uu c h 3 + g%uc c h 3 + glc c (ch 2 + th x ) 

+ g^t c (-ch 2 + thx) + gfdd c h 3 + glds c h 3 + gfs c (sh 2 + bhx) + g 2 b c (-sh 2 + bht) + H.c. (14) 
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FIG. 3. Different branching ratios involving the production and decay of h a . In all cases, m Xa — 20 GeV is assumed. 



We then rotate the scalars in the Yukawa Lagrangian to their physical basis {h a , hb, h c } which gives the Yukawa 
matrices Y{ a j, c y The individual mixing matrices for up- and down-type quarks contain large angles as a consequence 
of S3 symmetry and the particle assignments [5] . Specifically, the doublet representation of S3 generates maximal 
mixing when V\ — v-i- Now, the Cabibbo-Kobayashi-Maskawa matrix involves a relative alignment of those two 
matrices which yields small mixing for quarks. Similarly, the Pontecorvo-Maki-Nakagawa-Sakata matrix is given by 
the relative orientation of the mixing matrices for the charged leptons and neutrinos. Since the neutrino mass matrix 
generated in the present context by a type-II seesaw mechanism turns out to be diagonal, the large mixing angles in 
the lepton sector survive. There are two generic textures of Yukawa couplings in this model [7] : 



Ya = 



r 13 \ 
r 23 

Y 32 ) 



n, = 



(y u y 12 

Y21 Y 2 2 
V Y 33/ 



(15) 



Here Y a symbolically describes the Yukawa couplings for h a , Xa and h£, while Yf, jC describe the couplings for hb,h c , Xb 
and h£ . The pattern holds both for leptons and quarks [7] and reproduces the observed masses and mixing [5j . The 
off-diagonal couplings in Yf, c are numerically small and can be controlled by one free parameter which keeps processes 
like fi — ¥ e~/ and meson mixing well under control. The largest off-diagonal coupling in Y a is {h a /x a )ct which is about 
0.8; it leads to viable production channel of h a via t decays as described in the next section. The next largest couplings 
are {h a /Xa)sb « 0.02 and [h a / Xa) fJ-T ~ 0.008. The XaM T coupling induces an interesting decay channel potentially 
observable at the LHC. Note that since the h+ tb coupling does not exist the mass of h+ is not constrained by the 
LHC searches in the t — > h + b channel in the mass window of 80-160 GeV P^lfTB] , 



IV. OBSERVING h a AT THE LHC 

If kinematically allowed the dominant production of h a occurs through t — > h a c [Fig. [2|a)]. The subsequent decay 
channels depend crucially on the mass of the pseudoscalar Xa'- if m h a < m xai h a decays dominantly into b and s 
quarks, or r and fi [see Fig.[2jb)]. The branching ratio (BR) for t — > h a c is about 0.17(0.06) for m,h a = 130(150) GeV. 
Then h a — > [it proceeds with a BR of 10% and h a — > bs with 90%. 

A spectacular channel opens up when h a — > XaZ is kinematically accessible [Fig. |2jc)]. The BR of h a — > Zxa 
is almost 100% due to the numerical dominance of the gauge coupling over the Yukawa couplings involving light 
fermions, followed by Xa Tfi with a BR of ~ 10%, and a Z —> jiji BR of ~ 3%. If two h a are produced from tt 
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pairs, this could lead to a characteristic signal with up to six muons with the tau tags. The BRs for t — > ch a and 
subsequently h a — > XaZ — > Tfipfj, are plotted in Figs. [3]ja)-(c). For these plots m Xa = 20GeV has been assumed, 
which is allowed by current data. The BR peaks for irif la = 110 GeV once the kinematic threshold is crossed and then 
falls sharply for larger masses due to phase space constraints. 



V. CONCLUSIONS 



This is a natural extension of our previous work [7] where we studied only the scalar sector assuming the pseu- 
doscalars to be too heavy to be relevant. In this work we have analyzed the complete scalar/pseudoscalar sector of 
an S3 flavor model. We deal with three CP-even, two CP-odd and two sets of charged scalar particles. In this work 
we have improved our potential minimization technique which enabled us to explore a larger region of the allowed 
parameter space. It is possible to arrange the mass spectrum in full compatibility with the current LHC data, with the 
scalar hb mimicking the Higgs-like object lurking around 125 GeV. The specific scalar (pseudoscalar) with prominent 
non-standard gauge and Yukawa interactions, namely h a (xa)j evade standard searches at LEP/Tevatron/LHC and 
hence can be rather light. The other scalars/pseudoscalars can be arranged to stay beyond the current LHC reach 
(e.g., 550 GeV). In particular, we have identified a promising channel for h a search involving up to six muons in the 
final state with the tau tags. We urge our experimental colleagues to dig out this information which is probably 
buried in the existing data. 
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